Convolutional Neural Network (CNN) + VGG-16 base model + Data Augmentation

Image classification model

Josue Quiros Batista

---

Model 1: Convolutional Neural Network (CNN)

---

Model 2

VGG-16 base model:

• 16 layers with trainable weights: 13 convolutional layers as a powerful feature extractor and 3 fully-connected layers that make the final classification.
• pre-trained on the ImageNet dataset.

---

Model 3

VGG-16 base model + Feed Forward Neural Network (FFNN)

---

Model 4

VGG-16 base model + Feed Forward Neural Network (FFNN) + Data Augmentation To artificially increase the size and diversity of a training dataset.

---

Problem Statement

Business Context

Workplace safety in hazardous environments like construction sites and industrial plants is crucial to prevent accidents and injuries. One of the most important safety measures is ensuring workers wear safety helmets, which protect against head injuries from falling objects and machinery. Non-compliance with helmet regulations increases the risk of serious injuries or fatalities, making effective monitoring essential, especially in large-scale operations where manual oversight is prone to errors and inefficiency.

To overcome these challenges, SafeGuard Corp plans to develop an automated image analysis system capable of detecting whether workers are wearing safety helmets. This system will improve safety enforcement, ensuring compliance and reducing the risk of head injuries. By automating helmet monitoring, SafeGuard aims to enhance efficiency, scalability, and accuracy, ultimately fostering a safer work environment while minimizing human error in safety oversight.

Objective

As a data scientist at SafeGuard Corp, you are tasked with developing an image classification model that classifies images into one of two categories:

• With Helmet: Workers wearing safety helmets.
• Without Helmet: Workers not wearing safety helmets.

Data Description

The dataset consists of 631 images, equally divided into two categories:

• With Helmet: 311 images showing workers wearing helmets.
• Without Helmet: 320 images showing workers not wearing helmets.

Dataset Characteristics:

• Variations in Conditions: Images include diverse environments such as construction sites, factories, and industrial settings, with variations in lighting, angles, and worker postures to simulate real-world conditions.
• Worker Activities: Workers are depicted in different actions such as standing, using tools, or moving, ensuring robust model learning for various scenarios.

Used libraries

• numpy
• pandas
• seaborn
• matplotlib
• math
• random
• cv2
• keras
• tensorflow
• sklearn

Data Overview

Loading the data

# Loading the images file
_images = np.load('images_proj.npy')

# Loading the labels
_labels = pd.read_csv('Labels_proj.csv')

print("There are ", _images.shape, "images in the training data")
print("There are ", _labels.shape, "labels in the training data")

There are  (631, 200, 200, 3) images in the training data
There are  (631, 1) labels in the training data

Observations

• This dataset consists of 631 color images, each image having 3 channels (RGB)(3-dimensional Numpy array).
• The second and third dimensions 200, 200 denote the number of pixels.
• Every image is labeled.

Exploratory Data Analysis

Plot random images from each of the classes and print their corresponding labels.

Checking for class imbalance

ax = sns.countplot(x=_labels.iloc[:, 0]) # Getting first column count

plt.title("Distribution")
plt.xlabel("Class")
plt.ylabel("Count")
plt.xticks(ticks=[0, 1], labels=["Without Helmet", "With Helmet"])
ax.bar_label(ax.containers[0], fmt='%d')
plt.show()

Observations

• Both classes, "Without helmet" and "With helmet" are quite balanced. Each category has approximately the same number of images.

Data Preprocessing

Splitting the dataset

X = _images
y = _labels

# Splitting data into training, validation and test set
X_temp, X_test, y_temp, y_test = train_test_split(
    X, y, test_size=0.2, random_state=1, stratify=y, shuffle=True
)

# Split the temporary set into train and validation
X_train, X_val, y_train, y_val = train_test_split(
    X_temp, y_temp, test_size=0.2, random_state=1, stratify=y_temp, shuffle=True
)
print(X_train.shape, X_val.shape, X_test.shape)

(403, 200, 200, 3) (101, 200, 200, 3) (127, 200, 200, 3)

Data Normalization

X_train_normalized = X_train.astype('float32')/255.0
X_val_normalized = X_val.astype('float32')/255.0
X_test_normalized = X_test.astype('float32')/255.0

Observations

Pixel values range from 0-255. Divide all the pixel values by 255 to standardize the images to have values between 0-1.

Normalization:

• Reduces variation between the data points by placing all the data on the same scale.
• Would be less variation, which helps prevent exploding gradient problems, or vanishing gradient.
• Makes training a network more efficient and faster.

Model Building

Model Evaluation Criterion

"It would be more beneficial to reduce false positives. These are cases when the model predicts that a worker is using a helmet, but actually he or she is not. This may lead to an increase in undetected cases and consequently increase the likelihood of non-compliance with its use. Therefore, the evaluation criterion will be PRECISION, which is important when minimizing false positives is a priority

True Positives / (True Positives + False Positives)

Precision: gauges the ratio of true positive predictions to the total positive predictions made by the model.

_perf_training = pd.DataFrame()
_perf_validation = pd.DataFrame()

Model 1: Simple Convolutional Neural Network (CNN)

_m1 = Sequential()

# Feature Extraction layers: convolutional layers
# (403, 200, 200, 3) (101, 200, 200, 3) (127, 200, 200, 3)
_m1.add(Conv2D(32, (3, 3), activation='relu', input_shape=(200, 200, 3), padding="same"))
_m1.add(MaxPooling2D((3, 3), padding='same'))
# Second
_m1.add(Conv2D(32, (3, 3), activation='relu', padding="same"))
_m1.add(BatchNormalization())
_m1.add(MaxPooling2D((3, 3), padding='same'))
# Third
_m1.add(Conv2D(16, (3, 3), activation='relu', padding="same"))

# Flattening the output of the conv layer after max pooling to make it ready for creating dense connections
_m1.add(Flatten())
_m1.add(Dense(64, activation='relu'))
 # Binary classification
_m1.add(Dense(1, activation='sigmoid'))

# Compiling
# Precision
_opt = Adam(learning_rate=0.001)
_m1.compile(optimizer=_opt, loss='binary_crossentropy', metrics=["accuracy","Precision"])

# Summary
_m1.summary()

Model: "sequential_1"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ conv2d (Conv2D)                 │ (None, 200, 200, 32)   │           896 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ max_pooling2d (MaxPooling2D)    │ (None, 67, 67, 32)     │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ conv2d_1 (Conv2D)               │ (None, 67, 67, 32)     │         9,248 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ batch_normalization             │ (None, 67, 67, 32)     │           128 │
│ (BatchNormalization)            │                        │               │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ max_pooling2d_1 (MaxPooling2D)  │ (None, 23, 23, 32)     │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ conv2d_2 (Conv2D)               │ (None, 23, 23, 16)     │         4,624 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ flatten_1 (Flatten)             │ (None, 8464)           │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_3 (Dense)                 │ (None, 64)             │       541,760 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_4 (Dense)                 │ (None, 1)              │            65 │
└─────────────────────────────────┴────────────────────────┴───────────────┘

 Total params: 556,721 (2.12 MB)

 Trainable params: 556,657 (2.12 MB)

 Non-trainable params: 64 (256.00 B)

The Feature Extraction layers which are comprised of convolutional and pooling layers:

First Conv2D Layer Input shape to the layer is (200, 200, 3) - Images of size 200x200. Padding "same" in order to keep the output shape the same as that of the input shape. Number of Filters: 32 Size of the Filter: 3x3 Number of parameters: (3 x 3 x 1 + 1) x 32 = 320

Activation function: ReLu. Max Pooling Layer: pooling has no trainable parameters = 0.

Second Conv2D Layer Number of Filters: 32 Size of the Filter: 3x3

Activation function: ReLu. Batch normalization Max Pooling Layer: pooling has no trainable parameters = 0.

Third Conv2D Layer Number of Filters: 16 Size of the Filter: 3x3

Flatten

The Fully Connected classification layers for prediction.

Fully Connected Dense Layer Number of neurons: 64

Output Layer Number of neurons: 1 Activation: Sigmoid - binary classification

Fitting the model on the train data:

_m1_results = _m1.fit(
            X_train_normalized, y_train,
            epochs=15,
            validation_data=(X_val_normalized,y_val),
            shuffle=True,
            batch_size=16,
            verbose=2
)

Epoch 1/15
26/26 - 11s - 426ms/step - Precision: 0.8873 - accuracy: 0.9007 - loss: 0.2281 - val_Precision: 1.0000 - val_accuracy: 0.6436 - val_loss: 0.5888
Epoch 2/15
26/26 - 1s - 31ms/step - Precision: 0.9900 - accuracy: 0.9950 - loss: 0.0132 - val_Precision: 1.0000 - val_accuracy: 0.8911 - val_loss: 0.2589
Epoch 3/15
26/26 - 1s - 54ms/step - Precision: 0.9899 - accuracy: 0.9926 - loss: 0.0135 - val_Precision: 1.0000 - val_accuracy: 0.5941 - val_loss: 0.9445
Epoch 4/15
26/26 - 1s - 39ms/step - Precision: 0.9949 - accuracy: 0.9926 - loss: 0.0366 - val_Precision: 0.9074 - val_accuracy: 0.9406 - val_loss: 0.1646
Epoch 5/15
26/26 - 1s - 48ms/step - Precision: 0.9949 - accuracy: 0.9950 - loss: 0.0043 - val_Precision: 0.5882 - val_accuracy: 0.6535 - val_loss: 0.6261
Epoch 6/15
26/26 - 1s - 44ms/step - Precision: 0.9949 - accuracy: 0.9926 - loss: 0.0250 - val_Precision: 0.7812 - val_accuracy: 0.8614 - val_loss: 0.4192
Epoch 7/15
26/26 - 1s - 24ms/step - Precision: 0.9899 - accuracy: 0.9926 - loss: 0.0333 - val_Precision: 0.5618 - val_accuracy: 0.6139 - val_loss: 1.2890
Epoch 8/15
26/26 - 1s - 24ms/step - Precision: 0.9899 - accuracy: 0.9901 - loss: 0.0293 - val_Precision: 0.6410 - val_accuracy: 0.7228 - val_loss: 0.7901
Epoch 9/15
26/26 - 1s - 50ms/step - Precision: 0.9950 - accuracy: 0.9975 - loss: 0.0131 - val_Precision: 0.9737 - val_accuracy: 0.8614 - val_loss: 0.3811
Epoch 10/15
26/26 - 1s - 44ms/step - Precision: 0.9949 - accuracy: 0.9950 - loss: 0.0142 - val_Precision: 0.9783 - val_accuracy: 0.9406 - val_loss: 0.1623
Epoch 11/15
26/26 - 0s - 18ms/step - Precision: 1.0000 - accuracy: 1.0000 - loss: 1.1531e-04 - val_Precision: 0.9796 - val_accuracy: 0.9703 - val_loss: 0.1424
Epoch 12/15
26/26 - 1s - 23ms/step - Precision: 1.0000 - accuracy: 1.0000 - loss: 4.1390e-06 - val_Precision: 0.9796 - val_accuracy: 0.9703 - val_loss: 0.1463
Epoch 13/15
26/26 - 0s - 16ms/step - Precision: 1.0000 - accuracy: 1.0000 - loss: 4.0305e-06 - val_Precision: 0.9796 - val_accuracy: 0.9703 - val_loss: 0.1508
Epoch 14/15
26/26 - 1s - 24ms/step - Precision: 1.0000 - accuracy: 1.0000 - loss: 3.8918e-06 - val_Precision: 0.9800 - val_accuracy: 0.9802 - val_loss: 0.1548
Epoch 15/15
26/26 - 0s - 18ms/step - Precision: 1.0000 - accuracy: 1.0000 - loss: 3.7546e-06 - val_Precision: 0.9800 - val_accuracy: 0.9802 - val_loss: 0.1579

Performance check

plt.title('Training Loss versus Validation Loss - Model 0')
plt.plot(_m1_results.history['loss'], label='Training')
plt.plot(_m1_results.history['val_loss'], label='Validation')
plt.legend()
plt.show()

plt.plot(_m1_results.history['accuracy'])
plt.plot(_m1_results.history['val_accuracy'])
plt.title('Model Accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Validation'], loc='upper left')
plt.show()

score_m1 = _m1.evaluate(X_val_normalized, y_val)
score_m1

4/4 ━━━━━━━━━━━━━━━━━━━━ 1s 20ms/step - Precision: 0.9677 - accuracy: 0.9692 - loss: 0.2424 

[0.15787388384342194, 0.9801980257034302, 0.9800000190734863]

print(_m1_results.history.keys())

dict_keys(['Precision', 'accuracy', 'loss', 'val_Precision', 'val_accuracy', 'val_loss'])

plt.title('Training Precision versus Validation Precision - Model 1')
plt.plot(_m1_results.history['Precision'], label='Training')
plt.plot(_m1_results.history['val_Precision'], label='Validation')
plt.legend()
plt.show()

#Prediction on validation set
y_pred_train = _m1.predict(X_train_normalized)
y_pred_train = (y_pred_train > 0.5).astype(int)

y_pred_val = _m1.predict(X_val_normalized)
y_pred_val = (y_pred_val > 0.5).astype(int)

_perf_training = pd.DataFrame([{
    "Model": "Model 1 -  Simple Convolutional Neural Network (CNN)",
    "Precision": precision_score(y_train, y_pred_train)
}])

_perf_validation = pd.DataFrame([{
    "Model": "Model 1 -  Simple Convolutional Neural Network (CNN)",
    "Precision": precision_score(y_val, y_pred_val)
}])

13/13 ━━━━━━━━━━━━━━━━━━━━ 1s 29ms/step
4/4 ━━━━━━━━━━━━━━━━━━━━ 0s 62ms/step

_perf_training

	Model	Precision
0	Model 1 - Simple Convolutional Neural Network...	1.0

_perf_validation

	Model	Precision
0	Model 1 - Simple Convolutional Neural Network...	0.98

_m1_train_performance = model_performance_classification(_m1, X_train_normalized,y_train)
_m1_train_performance

13/13 ━━━━━━━━━━━━━━━━━━━━ 0s 16ms/step

	Accuracy	Recall	Precision	F1 Score
0	1.0	1.0	1.0	1.0

_m1_val_performance = model_performance_classification(_m1, X_val_normalized,y_val)
_m1_val_performance

4/4 ━━━━━━━━━━━━━━━━━━━━ 0s 17ms/step

	Accuracy	Recall	Precision	F1 Score
0	0.980198	0.980198	0.980198	0.980198

Observations

• During the early epochs, significant variations are observed in the values of val_accuracy and val_loss, which could suggest inconsistent, and therefore unstable, generalization.
• In the intermediate epochs, a potential overfitting is noted as the accuracy value on the validation set decreases. *During the final epochs, stable values are achieved, indicating adequate precision and accuracy.
• Attention should be paid to potential overfitting. It is possible that the model is memorizing data and noise.
• A possible data leakage was investigated, but no evidence of its occurrence was found.
• Model has perfomed well on the train and validation data, with a validation accuracy of 98%.

Confusion matrix

plot_confusion_matrix(_m1,X_train_normalized,y_train)

13/13 ━━━━━━━━━━━━━━━━━━━━ 0s 13ms/step

Observations

On train set

• 198 true positives were correctly predicted as true, and 205 true negatives were predicted as negative.

Accuracy: (205 + 198) / (205 + 198) = 1.0

Precision: 198 / (198 + 0) = 1.0

Recall: 198 / (198 + 0) = 1.0

F1-score: 1.0

• There are no false positives or false negatives.

plot_confusion_matrix(_m1,X_val_normalized,y_val)

4/4 ━━━━━━━━━━━━━━━━━━━━ 0s 13ms/step

The model generalizes well on the validation set. It makes only 2 classification errors (1 false positive and 1 false negative)

Vizualizing the predictions

_img_check_1 = random.randint(0, 100)
plt.imshow(X_val[_img_check_1])
plt.show()

_real = y_val.iloc[_img_check_1]
print('Real value for image ', _img_check_1, ": ", _real)

_predicted = _m1.predict(X_val_normalized[_img_check_1].reshape(1,200,200,3))
_pred_label = _predicted[0][0]>0.5
print('Predicted:', 1 if _pred_label else 0)

Real value for image  37 :  Label    1
Name: 569, dtype: int64
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 202ms/step
Predicted: 1

_img_check_2 = random.randint(0, 100)
plt.imshow(X_val[_img_check_2])
plt.show()

_real = y_val.iloc[_img_check_2]
print('Real value for image ', _img_check_2, ": ", _real)

_predicted = _m1.predict(X_val_normalized[_img_check_2].reshape(1,200,200,3))
_pred_label = _predicted[0][0]>0.5
print('Predicted:', 1 if _pred_label else 0)

Real value for image  45 :  Label    1
Name: 583, dtype: int64
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 30ms/step
Predicted: 1

_img_check_3 = random.randint(0, 100)
plt.imshow(X_val[_img_check_3])
plt.show()

_real = y_val.iloc[_img_check_3]
print('Real value for image ', _img_check_3, ": ", _real)

_predicted = _m1.predict(X_val_normalized[_img_check_3].reshape(1,200,200,3))
_pred_label = _predicted[0][0]>0.5
print('Predicted:', 1 if _pred_label else 0)

Real value for image  82 :  Label    1
Name: 43, dtype: int64
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 36ms/step
Predicted: 1

Observations:

• The 3 randomly selected images were predicted correctly.

Model 2: (VGG-16 (Base))

Clear the previous model's history from the session

# Clearing backend
from tensorflow.keras import backend
backend.clear_session()

Fix the seed after clearing the backend.

# Fixing the seed for random number generators
import random
np.random.seed(1)
random.seed(1)
tf.random.set_seed(1)

_VGG = VGG16(weights='imagenet', include_top = False, input_shape = (224,224,3))

# Non-trainable layers
for layer in _VGG.layers:
    layer.trainable = False

_m2 = Sequential()
# Add convolutional
_m2.add(_VGG)

# Flattening the output of the VGG model
_m2.add(Flatten())

# Adding a dense output layer
_m2.add(Dense(1, activation='sigmoid'))

# Compiling
_m2.compile(optimizer=Adam(), loss='binary_crossentropy', metrics=["accuracy","Precision"])
_m2.summary()

Model: "sequential"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ vgg16 (Functional)              │ (None, 7, 7, 512)      │    14,714,688 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ flatten (Flatten)               │ (None, 25088)          │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense (Dense)                   │ (None, 1)              │        25,089 │
└─────────────────────────────────┴────────────────────────┴───────────────┘

 Total params: 14,739,777 (56.23 MB)

 Trainable params: 25,089 (98.00 KB)

 Non-trainable params: 14,714,688 (56.13 MB)

• Due to the utilization of the VGG16 architecture for transfer learning, pre-trained on the ImageNet dataset, the images will be reshaped to tha appropriate size expected by the model: 224, 224, 3.
• The convolutional and pooling layers will be used; therefore, their weights will be frozen, meaning no training instructions will be executed upon them.
• It was specified "include_top = False", then the model will be imported without the fully-connected layers.

X_train_resized = tf.image.resize(X_train_normalized, [224, 224])
X_val_resized = tf.image.resize(X_val_normalized, [224, 224])

train_datagen =  ImageDataGenerator()
train_generator = train_datagen.flow(X_train_resized, y_train,
                                     batch_size=16, seed=1, shuffle=True)

Observations:

• The model will now use the normalized and resized dataset. Additionally, a batch size of 16 is specified, and it will run with shuffle set to true.

## Fitting the VGG model
_m2_results = _m2.fit(train_generator, epochs=15, validation_data=(X_val_resized, y_val),
                      verbose=2)

Epoch 1/15
26/26 - 6s - 248ms/step - Precision: 0.9538 - accuracy: 0.9479 - loss: 0.1269 - val_Precision: 0.9804 - val_accuracy: 0.9901 - val_loss: 0.0132
Epoch 2/15
26/26 - 2s - 90ms/step - Precision: 1.0000 - accuracy: 1.0000 - loss: 0.0023 - val_Precision: 0.9804 - val_accuracy: 0.9901 - val_loss: 0.0137
Epoch 3/15
26/26 - 2s - 83ms/step - Precision: 1.0000 - accuracy: 1.0000 - loss: 0.0012 - val_Precision: 0.9804 - val_accuracy: 0.9901 - val_loss: 0.0100
Epoch 4/15
26/26 - 2s - 90ms/step - Precision: 1.0000 - accuracy: 1.0000 - loss: 9.7656e-04 - val_Precision: 0.9804 - val_accuracy: 0.9901 - val_loss: 0.0101
Epoch 5/15
26/26 - 2s - 90ms/step - Precision: 1.0000 - accuracy: 1.0000 - loss: 8.5377e-04 - val_Precision: 0.9804 - val_accuracy: 0.9901 - val_loss: 0.0099
Epoch 6/15
26/26 - 3s - 98ms/step - Precision: 1.0000 - accuracy: 1.0000 - loss: 7.4949e-04 - val_Precision: 0.9804 - val_accuracy: 0.9901 - val_loss: 0.0096
Epoch 7/15
26/26 - 2s - 84ms/step - Precision: 1.0000 - accuracy: 1.0000 - loss: 6.7190e-04 - val_Precision: 0.9804 - val_accuracy: 0.9901 - val_loss: 0.0092
Epoch 8/15
26/26 - 2s - 84ms/step - Precision: 1.0000 - accuracy: 1.0000 - loss: 6.0065e-04 - val_Precision: 0.9804 - val_accuracy: 0.9901 - val_loss: 0.0093
Epoch 9/15
26/26 - 2s - 90ms/step - Precision: 1.0000 - accuracy: 1.0000 - loss: 5.3734e-04 - val_Precision: 0.9804 - val_accuracy: 0.9901 - val_loss: 0.0090
Epoch 10/15
26/26 - 2s - 91ms/step - Precision: 1.0000 - accuracy: 1.0000 - loss: 4.8772e-04 - val_Precision: 0.9804 - val_accuracy: 0.9901 - val_loss: 0.0088
Epoch 11/15
26/26 - 3s - 105ms/step - Precision: 1.0000 - accuracy: 1.0000 - loss: 4.4523e-04 - val_Precision: 0.9804 - val_accuracy: 0.9901 - val_loss: 0.0087
Epoch 12/15
26/26 - 2s - 90ms/step - Precision: 1.0000 - accuracy: 1.0000 - loss: 4.0581e-04 - val_Precision: 0.9804 - val_accuracy: 0.9901 - val_loss: 0.0086
Epoch 13/15
26/26 - 2s - 82ms/step - Precision: 1.0000 - accuracy: 1.0000 - loss: 3.7183e-04 - val_Precision: 0.9804 - val_accuracy: 0.9901 - val_loss: 0.0082
Epoch 14/15
26/26 - 2s - 90ms/step - Precision: 1.0000 - accuracy: 1.0000 - loss: 3.4271e-04 - val_Precision: 0.9804 - val_accuracy: 0.9901 - val_loss: 0.0082
Epoch 15/15
26/26 - 2s - 90ms/step - Precision: 1.0000 - accuracy: 1.0000 - loss: 3.1344e-04 - val_Precision: 0.9804 - val_accuracy: 0.9901 - val_loss: 0.0080

Observations:

• Both the training loss and validation loss are extremely low.
• val_accuracy and val_precision exhibit stability.
• There are no significant fluctuations indicating instability or noise.
• The validation performance is stable and reliable.

Performance check

plt.title('Training Loss versus Validation Loss - Model 2')
plt.plot(_m2_results.history['loss'], label='Training')
plt.plot(_m2_results.history['val_loss'], label='Validation')
plt.legend()
plt.show()

plt.plot(_m2_results.history['accuracy'])
plt.plot(_m2_results.history['val_accuracy'])
plt.title('Model Accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Validation'], loc='upper left')
plt.show()

score_m2 = _m2.evaluate(X_val_resized, y_val)
score_m2

4/4 ━━━━━━━━━━━━━━━━━━━━ 0s 110ms/step - Precision: 0.9689 - accuracy: 0.9846 - loss: 0.0122

[0.008016041480004787, 0.9900990128517151, 0.9803921580314636]

#Prediction on validation set
y_pred_train = _m2.predict(X_train_resized)
y_pred_train = (y_pred_train > 0.5).astype(int)

13/13 ━━━━━━━━━━━━━━━━━━━━ 3s 164ms/step

y_pred_val = _m2.predict(X_val_resized)
y_pred_val = (y_pred_val > 0.5).astype(int)

4/4 ━━━━━━━━━━━━━━━━━━━━ 1s 165ms/step

_perf_training = pd.concat([_perf_training, pd.DataFrame([{
    "Model": "Model 2 - (VGG-16 (Base))",
    "Precision": precision_score(y_train, y_pred_train)
}])], ignore_index=True)

_perf_validation = pd.concat([_perf_validation,pd.DataFrame([{
    "Model": "Model 2 - (VGG-16 (Base))",
    "Precision": precision_score(y_val, y_pred_val)
}])], ignore_index=True)

_perf_training

	Model	Precision
0	Model 1 - Simple Convolutional Neural Network...	1.0
1	Model 2 - (VGG-16 (Base))	1.0

_perf_validation

	Model	Precision
0	Model 1 - Simple Convolutional Neural Network...	0.980000
1	Model 2 - (VGG-16 (Base))	0.980392

_m2_train_performance = model_performance_classification(_m2, X_train_resized,y_train)
_m2_train_performance

13/13 ━━━━━━━━━━━━━━━━━━━━ 2s 127ms/step

	Accuracy	Recall	Precision	F1 Score
0	1.0	1.0	1.0	1.0

_m2_val_performance = model_performance_classification(_m2, X_val_resized,y_val)
_m2_val_performance

4/4 ━━━━━━━━━━━━━━━━━━━━ 0s 99ms/step 

	Accuracy	Recall	Precision	F1 Score
0	0.990099	0.990099	0.990293	0.990099

Confusion matrix

plot_confusion_matrix(_m2,X_train_resized,y_train)

13/13 ━━━━━━━━━━━━━━━━━━━━ 2s 126ms/step

Observations:

• Both confusion matrices yield the same results as model one (Simple Convolutional Neural Network).

plot_confusion_matrix(_m2,X_val_resized,y_val)

4/4 ━━━━━━━━━━━━━━━━━━━━ 0s 99ms/step 

Visualizing the prediction:

_img_check_1 = random.randint(0, 100)
plt.imshow(X_val[_img_check_1])
plt.show()

_real = y_val.iloc[_img_check_1]
print('Real value for image ', _img_check_1, ": ", _real)

Real value for image  14 :  Label    0
Name: 602, dtype: int64

_predicted = _m2.predict(X_val_resized[_img_check_1:_img_check_1+1])
_pred_label = _predicted[0][0] > 0.5
print('Predicted:', 1 if _pred_label else 0)

1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 370ms/step
Predicted: 0

_img_check_2 = random.randint(0, 100)
plt.imshow(X_val[_img_check_2])
plt.show()

_real = y_val.iloc[_img_check_2]
print('Real value for image ', _img_check_2, ": ", _real)
_predicted = _m2.predict(X_val_resized[_img_check_2:_img_check_2+1])
_pred_label = _predicted[0][0]>0.5
print('Predicted:', 1 if _pred_label else 0)

Real value for image  32 :  Label    1
Name: 263, dtype: int64
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 49ms/step
Predicted: 1

_img_check_3 = random.randint(0, 100)
plt.imshow(X_val[_img_check_3])
plt.show()

_real = y_val.iloc[_img_check_3]
print('Real value for image ', _img_check_3, ": ", _real)

_predicted = _m2.predict(X_val_resized[_img_check_3:_img_check_3+1])
_pred_label = _predicted[0][0]>0.5
print('Predicted:', 1 if _pred_label else 0)

Real value for image  27 :  Label    1
Name: 207, dtype: int64
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 69ms/step
Predicted: 1

Observations:

• The 3 randomly selected images were predicted correctly.

Model 3: (VGG-16 (Base + FFNN))

# Clearing backend
from tensorflow.keras import backend
backend.clear_session()

# Fixing the seed for random number generators
import random
np.random.seed(1)
random.seed(1)
tf.random.set_seed(1)

_m3 = Sequential()
# Add convolutional
_m3.add(_VGG)

# Flattening the output of the VGG model
_m3.add(Flatten())

# FFNN - Feed Forward Neural Network
_m3.add(Dense(64, activation='relu'))
_m3.add(Dropout(0.3))
_m3.add(Dense(16, activation='relu'))

# Binary classification
_m3.add(Dense(1, activation='sigmoid'))

# Compiling
_Adam = tf.keras.optimizers.Adam()
_m3.compile(optimizer=_Adam, loss='binary_crossentropy', metrics=["accuracy","Precision"])
_m3.summary()

Model: "sequential"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ vgg16 (Functional)              │ (None, 7, 7, 512)      │    14,714,688 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ flatten (Flatten)               │ (None, 25088)          │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense (Dense)                   │ (None, 64)             │     1,605,696 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout (Dropout)               │ (None, 64)             │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_1 (Dense)                 │ (None, 16)             │         1,040 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_2 (Dense)                 │ (None, 1)              │            17 │
└─────────────────────────────────┴────────────────────────┴───────────────┘

 Total params: 16,321,441 (62.26 MB)

 Trainable params: 1,606,753 (6.13 MB)

 Non-trainable params: 14,714,688 (56.13 MB)

Observations:

• For this subsequent model, the base of the previous Model 2 is retained; however, several Feed Forward Neural Network layers are added. One layer consists of 64 neurons, and another with 16, both utilizing the ReLU activation function.
• Additionally, a dropout rate of 0.3 is incorporated.
• The Adam optimizer is employed, with no specific learning rate defined for this initial model. Finally, the metrics considered for evaluation are accuracy and precision.

## Fitting the VGG model
_m3_results = _m3.fit(train_generator, epochs=15, validation_data=(X_val_resized, y_val),
                      verbose=2)

Epoch 1/15
26/26 - 8s - 320ms/step - Precision: 0.8378 - accuracy: 0.8809 - loss: 0.3719 - val_Precision: 0.9804 - val_accuracy: 0.9901 - val_loss: 0.0129
Epoch 2/15
26/26 - 2s - 89ms/step - Precision: 1.0000 - accuracy: 0.9975 - loss: 0.0132 - val_Precision: 0.9804 - val_accuracy: 0.9901 - val_loss: 0.0224
Epoch 3/15
26/26 - 2s - 82ms/step - Precision: 1.0000 - accuracy: 1.0000 - loss: 0.0031 - val_Precision: 0.9804 - val_accuracy: 0.9901 - val_loss: 0.0242
Epoch 4/15
26/26 - 2s - 89ms/step - Precision: 1.0000 - accuracy: 1.0000 - loss: 0.0023 - val_Precision: 0.9804 - val_accuracy: 0.9901 - val_loss: 0.0289
Epoch 5/15
26/26 - 2s - 82ms/step - Precision: 1.0000 - accuracy: 0.9975 - loss: 0.0070 - val_Precision: 0.9804 - val_accuracy: 0.9901 - val_loss: 0.0298
Epoch 6/15
26/26 - 2s - 89ms/step - Precision: 1.0000 - accuracy: 0.9975 - loss: 0.0108 - val_Precision: 0.9804 - val_accuracy: 0.9901 - val_loss: 0.0445
Epoch 7/15
26/26 - 2s - 89ms/step - Precision: 1.0000 - accuracy: 1.0000 - loss: 0.0027 - val_Precision: 0.9804 - val_accuracy: 0.9901 - val_loss: 0.0239
Epoch 8/15
26/26 - 2s - 90ms/step - Precision: 1.0000 - accuracy: 1.0000 - loss: 0.0036 - val_Precision: 0.9804 - val_accuracy: 0.9901 - val_loss: 0.0314
Epoch 9/15
26/26 - 2s - 89ms/step - Precision: 1.0000 - accuracy: 0.9975 - loss: 0.0098 - val_Precision: 0.9804 - val_accuracy: 0.9901 - val_loss: 0.0409
Epoch 10/15
26/26 - 2s - 82ms/step - Precision: 1.0000 - accuracy: 1.0000 - loss: 0.0019 - val_Precision: 0.9804 - val_accuracy: 0.9901 - val_loss: 0.0356
Epoch 11/15
26/26 - 2s - 90ms/step - Precision: 1.0000 - accuracy: 0.9975 - loss: 0.0053 - val_Precision: 0.9804 - val_accuracy: 0.9901 - val_loss: 0.0364
Epoch 12/15
26/26 - 2s - 90ms/step - Precision: 1.0000 - accuracy: 1.0000 - loss: 0.0013 - val_Precision: 0.9804 - val_accuracy: 0.9901 - val_loss: 0.0324
Epoch 13/15
26/26 - 2s - 91ms/step - Precision: 1.0000 - accuracy: 1.0000 - loss: 0.0029 - val_Precision: 0.9804 - val_accuracy: 0.9901 - val_loss: 0.0372
Epoch 14/15
26/26 - 2s - 83ms/step - Precision: 0.9950 - accuracy: 0.9975 - loss: 0.0034 - val_Precision: 0.9804 - val_accuracy: 0.9901 - val_loss: 0.0298
Epoch 15/15
26/26 - 2s - 91ms/step - Precision: 1.0000 - accuracy: 1.0000 - loss: 0.0014 - val_Precision: 0.9804 - val_accuracy: 0.9901 - val_loss: 0.0379

Observations:

• The model correctly classifies 99% of the validation set, with stable val_accuracy values; however, the increasing val_loss could suggest signs of overfitting.
• The constant precision despite continued training may indicate that this model has reached its generalization limit with its current architecture.
• It would be advisable to incorporate measures to assess and mitigate the effects of overfitting.

Performance check

plt.title('Training Loss versus Validation Loss - Model 3')
plt.plot(_m3_results.history['loss'], label='Training')
plt.plot(_m3_results.history['val_loss'], label='Validation')
plt.legend()
plt.show()

plt.plot(_m3_results.history['accuracy'])
plt.plot(_m3_results.history['val_accuracy'])
plt.title('Model Accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Validation'], loc='upper left')
plt.show()

score_m3 = _m3.evaluate(X_val_resized, y_val)
score_m3

4/4 ━━━━━━━━━━━━━━━━━━━━ 0s 104ms/step - Precision: 0.9689 - accuracy: 0.9846 - loss: 0.0589

[0.037885893136262894, 0.9900990128517151, 0.9803921580314636]

#Prediction on validation set
y_pred_train = _m3.predict(X_train_resized)
y_pred_train = (y_pred_train > 0.5).astype(int)

y_pred_val = _m3.predict(X_val_resized)
y_pred_val = (y_pred_val > 0.5).astype(int)

_perf_training = pd.concat([_perf_training,pd.DataFrame([{
    "Model": "Model 3 - Model 3: (VGG-16 (Base + FFNN))",
    "Precision": precision_score(y_train, y_pred_train)
}])], ignore_index=True)

_perf_validation = pd.concat([_perf_validation, pd.DataFrame([{
    "Model": "Model 3: (VGG-16 (Base + FFNN))",
    "Precision": precision_score(y_val, y_pred_val)
}])], ignore_index=True)

13/13 ━━━━━━━━━━━━━━━━━━━━ 3s 164ms/step
4/4 ━━━━━━━━━━━━━━━━━━━━ 1s 163ms/step

_perf_training

	Model	Precision
0	Model 1 - Simple Convolutional Neural Network...	1.0
1	Model 2 - (VGG-16 (Base))	1.0
2	Model 3 - Model 3: (VGG-16 (Base + FFNN))	1.0

_perf_validation

	Model	Precision
0	Model 1 - Simple Convolutional Neural Network...	0.980000
1	Model 2 - (VGG-16 (Base))	0.980392
2	Model 3: (VGG-16 (Base + FFNN))	0.980392

_m3_train_performance = model_performance_classification(_m3, X_train_resized,y_train)
_m3_train_performance

13/13 ━━━━━━━━━━━━━━━━━━━━ 2s 136ms/step

	Accuracy	Recall	Precision	F1 Score
0	1.0	1.0	1.0	1.0

_m3_val_performance = model_performance_classification(_m3, X_val_resized,y_val)
_m3_val_performance

4/4 ━━━━━━━━━━━━━━━━━━━━ 0s 100ms/step

	Accuracy	Recall	Precision	F1 Score
0	0.990099	0.990099	0.990293	0.990099

Confusion matrix

plot_confusion_matrix(_m3,X_train_resized,y_train)

13/13 ━━━━━━━━━━━━━━━━━━━━ 2s 129ms/step

plot_confusion_matrix(_m3,X_val_resized,y_val)

4/4 ━━━━━━━━━━━━━━━━━━━━ 0s 102ms/step

Visualizing the predictions

_img_check_1 = random.randint(0, 100)
plt.imshow(X_val[_img_check_1])
plt.show()

_real = y_val.iloc[_img_check_1]
print('Real value for image ', _img_check_1, ": ", _real)

_predicted = _m3.predict(X_val_resized[_img_check_1:_img_check_1+1])
_pred_label = _predicted[0][0]>0.5
print('Predicted:', 1 if _pred_label else 0)

Real value for image  14 :  Label    0
Name: 602, dtype: int64
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 249ms/step
Predicted: 0

_img_check_2 = random.randint(0, 100)
plt.imshow(X_val[_img_check_2])
plt.show()

_real = y_val.iloc[_img_check_2]
print('Real value for image ', _img_check_2, ": ", _real)

_predicted = _m3.predict(X_val_resized[_img_check_2:_img_check_2+1])
_pred_label = _predicted[0][0]>0.5
print('Predicted:', 1 if _pred_label else 0)

Real value for image  32 :  Label    1
Name: 263, dtype: int64
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 38ms/step
Predicted: 1

_img_check_3 = random.randint(0, 100)
plt.imshow(X_val[_img_check_3])
plt.show()

_real = y_val.iloc[_img_check_3]
print('Real value for image ', _img_check_3, ": ", _real)

_predicted = _m3.predict(X_val_resized[_img_check_3:_img_check_3+1])
_pred_label = _predicted[0][0]>0.5
print('Predicted:', 1 if _pred_label else 0)

Real value for image  27 :  Label    1
Name: 207, dtype: int64
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 39ms/step
Predicted: 1

Observations:

• The 3 randomly selected images were predicted correctly.

Model 4: (VGG-16 (Base + FFNN + Data Augmentation)

• In most of the real-world case studies, it is challenging to acquire a large number of images and then train CNNs.

• To overcome this problem, one approach we might consider is Data Augmentation.

• CNNs have the property of translational invariance, which means they can recognise an object even if its appearance shifts translationally in some way. - Taking this attribute into account, we can augment the images using the techniques listed below

  • Horizontal Flip (should be set to True/False)
  • Vertical Flip (should be set to True/False)
  • Height Shift (should be between 0 and 1)
  • Width Shift (should be between 0 and 1)
  • Rotation (should be between 0 and 180)
  • Shear (should be between 0 and 1)
  • Zoom (should be between 0 and 1) etc.

Data augmentation should not be used in the validation/test data set.

Clear the previous model's history from the session and fix the seed after clearing the backend.

# Clearing backend
backend.clear_session()

# Fixing the seed for random number generators
import random
np.random.seed(1)
random.seed(1)
tf.random.set_seed(1)

_m4 = Sequential()
# Add convolutional
_m4.add(_VGG)

# Flattening the output of the VGG model
_m4.add(Flatten())

# FFNN - Feed Forward Neural Network
_m4.add(Dense(256, activation='relu'))
_m4.add(Dropout(0.4))
_m4.add(Dense(128, activation='relu'))

# Binary classification
_m4.add(Dense(1, activation='sigmoid'))

# Compiling
_Adam = tf.keras.optimizers.Adam(learning_rate=0.001)
_m4.compile(optimizer=_Adam, loss='binary_crossentropy', metrics=["accuracy","Precision"])
_m4.summary()

Model: "sequential"

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ vgg16 (Functional)              │ (None, 7, 7, 512)      │    14,714,688 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ flatten (Flatten)               │ (None, 25088)          │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense (Dense)                   │ (None, 256)            │     6,422,784 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dropout (Dropout)               │ (None, 256)            │             0 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_1 (Dense)                 │ (None, 128)            │        32,896 │
├─────────────────────────────────┼────────────────────────┼───────────────┤
│ dense_2 (Dense)                 │ (None, 1)              │           129 │
└─────────────────────────────────┴────────────────────────┴───────────────┘

 Total params: 21,170,497 (80.76 MB)

 Trainable params: 6,455,809 (24.63 MB)

 Non-trainable params: 14,714,688 (56.13 MB)

Data augmentation

train_datagen = ImageDataGenerator(horizontal_flip = True,
                                   vertical_flip = False,
                                   height_shift_range= 0.1,
                                   width_shift_range=0.1,
                                   rotation_range=20,
                                   shear_range = 0.2,
                                   zoom_range=0.2,
                                   fill_mode='nearest')

# Flowing training images in batches of 16 using train_datagen generator
train_generator = train_datagen.flow(X_train_resized, y_train,
                                                    batch_size=16, seed=1, shuffle=True)

• For this iteration, the base architecture of the preceding two models is retained. However, the following modifications have been implemented:

  • a. The Feed Forward Neural Network (FFNN) layers now comprise 256 and 128 neurons, respectively.
  • b. The dropout rate has been increased to 0.4.
  • c. A learning rate of 0.001 has been added to the Adam optimizer.
  • d. Data augmentation has been incorporated, with the above specifications.

## Fitting the VGG model
_m4_results = _m4.fit(train_generator, epochs=15, validation_data=(X_val_resized, y_val),
                      verbose=2)

Epoch 1/15
26/26 - 11s - 440ms/step - Precision: 0.9150 - accuracy: 0.9206 - loss: 0.1984 - val_Precision: 0.9804 - val_accuracy: 0.9901 - val_loss: 0.0677
Epoch 2/15
26/26 - 5s - 198ms/step - Precision: 0.9949 - accuracy: 0.9950 - loss: 0.0186 - val_Precision: 0.9804 - val_accuracy: 0.9901 - val_loss: 0.0610
Epoch 3/15
26/26 - 6s - 234ms/step - Precision: 1.0000 - accuracy: 1.0000 - loss: 0.0033 - val_Precision: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.0012
Epoch 4/15
26/26 - 5s - 196ms/step - Precision: 0.9848 - accuracy: 0.9826 - loss: 0.0713 - val_Precision: 0.9804 - val_accuracy: 0.9901 - val_loss: 0.0736
Epoch 5/15
26/26 - 6s - 234ms/step - Precision: 0.9849 - accuracy: 0.9876 - loss: 0.0547 - val_Precision: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.0040
Epoch 6/15
26/26 - 5s - 191ms/step - Precision: 0.9950 - accuracy: 0.9975 - loss: 0.0035 - val_Precision: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.0034
Epoch 7/15
26/26 - 6s - 232ms/step - Precision: 0.9950 - accuracy: 0.9975 - loss: 0.0028 - val_Precision: 1.0000 - val_accuracy: 1.0000 - val_loss: 3.4767e-04
Epoch 8/15
26/26 - 5s - 200ms/step - Precision: 0.9949 - accuracy: 0.9950 - loss: 0.0149 - val_Precision: 1.0000 - val_accuracy: 1.0000 - val_loss: 3.8455e-06
Epoch 9/15
26/26 - 6s - 216ms/step - Precision: 1.0000 - accuracy: 1.0000 - loss: 0.0015 - val_Precision: 1.0000 - val_accuracy: 1.0000 - val_loss: 1.3269e-05
Epoch 10/15
26/26 - 5s - 207ms/step - Precision: 1.0000 - accuracy: 1.0000 - loss: 1.9605e-05 - val_Precision: 1.0000 - val_accuracy: 1.0000 - val_loss: 1.1121e-04
Epoch 11/15
26/26 - 5s - 200ms/step - Precision: 1.0000 - accuracy: 1.0000 - loss: 3.6677e-05 - val_Precision: 1.0000 - val_accuracy: 1.0000 - val_loss: 8.3359e-05
Epoch 12/15
26/26 - 6s - 226ms/step - Precision: 1.0000 - accuracy: 1.0000 - loss: 1.1147e-06 - val_Precision: 1.0000 - val_accuracy: 1.0000 - val_loss: 8.0329e-05
Epoch 13/15
26/26 - 5s - 192ms/step - Precision: 0.9949 - accuracy: 0.9950 - loss: 0.0060 - val_Precision: 1.0000 - val_accuracy: 1.0000 - val_loss: 0.0014
Epoch 14/15
26/26 - 6s - 236ms/step - Precision: 1.0000 - accuracy: 1.0000 - loss: 1.1169e-06 - val_Precision: 1.0000 - val_accuracy: 1.0000 - val_loss: 2.7920e-05
Epoch 15/15
26/26 - 5s - 204ms/step - Precision: 1.0000 - accuracy: 1.0000 - loss: 2.0358e-06 - val_Precision: 1.0000 - val_accuracy: 1.0000 - val_loss: 2.1428e-05

Performance check

plt.title('Training Loss versus Validation Loss - Model 4')
plt.plot(_m4_results.history['loss'], label='Training')
plt.plot(_m4_results.history['val_loss'], label='Validation')
plt.legend()
plt.show()

plt.plot(_m4_results.history['accuracy'])
plt.plot(_m4_results.history['val_accuracy'])
plt.title('Model Accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Validation'], loc='upper left')
plt.show()

Observations

• The val_loss is consistently very low, showing no evident signs of deterioration or overfitting.
• Training is stable with no large fluctuations.
• Again sought to confirm the absence of data leakage, but its presence could not be identified.

score_m4 = _m4.evaluate(X_val_resized, y_val)
score_m4

4/4 ━━━━━━━━━━━━━━━━━━━━ 0s 103ms/step - Precision: 1.0000 - accuracy: 1.0000 - loss: 3.3370e-05

[2.1428091713460162e-05, 1.0, 1.0]

#Prediction on validation set
y_pred_train = _m4.predict(X_train_resized)
y_pred_train = (y_pred_train > 0.5).astype(int)

y_pred_val = _m4.predict(X_val_resized)
y_pred_val = (y_pred_val > 0.5).astype(int)

_perf_training = pd.concat([_perf_training, pd.DataFrame([{
    "Model": "Model 4 - (VGG-16 (Base + FFNN + Data Augmentation)",
    "Precision": precision_score(y_train, y_pred_train)
}])], ignore_index=True)

_perf_validation = pd.concat([_perf_validation, pd.DataFrame([{
    "Model": "Model 4 -  Simple Convolutional Neural Network (CNN)",
    "Precision": precision_score(y_val, y_pred_val)
}])], ignore_index=True)

13/13 ━━━━━━━━━━━━━━━━━━━━ 3s 162ms/step
4/4 ━━━━━━━━━━━━━━━━━━━━ 1s 163ms/step

_perf_training

	Model	Precision
0	Model 1 - Simple Convolutional Neural Network...	1.0
1	Model 2 - (VGG-16 (Base))	1.0
2	Model 3 - Model 3: (VGG-16 (Base + FFNN))	1.0
3	Model 4 - (VGG-16 (Base + FFNN + Data Augmenta...	1.0

_perf_validation

	Model	Precision
0	Model 1 - Simple Convolutional Neural Network...	0.980000
1	Model 2 - (VGG-16 (Base))	0.980392
2	Model 3: (VGG-16 (Base + FFNN))	0.980392
3	Model 4 - Simple Convolutional Neural Network...	1.000000

_m4_train_performance = model_performance_classification(_m4, X_train_resized,y_train)
_m4_train_performance

13/13 ━━━━━━━━━━━━━━━━━━━━ 2s 127ms/step

	Accuracy	Recall	Precision	F1 Score
0	1.0	1.0	1.0	1.0

_m4_val_performance = model_performance_classification(_m4, X_val_resized,y_val)
_m4_val_performance

4/4 ━━━━━━━━━━━━━━━━━━━━ 0s 100ms/step

	Accuracy	Recall	Precision	F1 Score
0	1.0	1.0	1.0	1.0

Confusion matrix

plot_confusion_matrix(_m4,X_train_resized,y_train)

13/13 ━━━━━━━━━━━━━━━━━━━━ 2s 131ms/step

plot_confusion_matrix(_m4,X_val_resized,y_val)

4/4 ━━━━━━━━━━━━━━━━━━━━ 0s 99ms/step 

Visualizing the predictions

_img_check_1 = random.randint(0, 100)
plt.imshow(X_val[_img_check_1])
plt.show()

_real = y_val.iloc[_img_check_1]
print('Real value for image ', _img_check_1, ": ", _real)

_predicted = _m4.predict(X_val_resized[_img_check_1:_img_check_1+1])
_pred_label = _predicted[0][0]>0.5
print('Predicted:', 1 if _pred_label else 0)

Real value for image  14 :  Label    0
Name: 602, dtype: int64
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 283ms/step
Predicted: 0

_img_check_2 = random.randint(0, 100)
plt.imshow(X_val[_img_check_2])
plt.show()

_real = y_val.iloc[_img_check_2]
print('Real value for image ', _img_check_2, ": ", _real)

_predicted = _m4.predict(X_val_resized[_img_check_2:_img_check_2+1])
_pred_label = _predicted[0][0]>0.5
print('Predicted:', 1 if _pred_label else 0)

Real value for image  32 :  Label    1
Name: 263, dtype: int64
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 35ms/step
Predicted: 1

_img_check_3 = random.randint(0, 100)
plt.imshow(X_val[_img_check_3])
plt.show()

_real = y_val.iloc[_img_check_3]
print('Real value for image ', _img_check_3, ": ", _real)

_predicted = _m4.predict(X_val_resized[_img_check_3:_img_check_3+1])
_pred_label = _predicted[0][0]>0.5
print('Predicted:', 1 if _pred_label else 0)

Real value for image  27 :  Label    1
Name: 207, dtype: int64
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 36ms/step
Predicted: 1

Observations

• All 3 randomly chosen images were predicted correctly.

Model Performance Comparison and Final Model Selection

_perf_training

	Model	Precision
0	Model 1 - Simple Convolutional Neural Network...	1.0
1	Model 2 - (VGG-16 (Base))	1.0
2	Model 3 - Model 3: (VGG-16 (Base + FFNN))	1.0
3	Model 4 - (VGG-16 (Base + FFNN + Data Augmenta...	1.0

_perf_validation

	Model	Precision
0	Model 1 - Simple Convolutional Neural Network...	0.980000
1	Model 2 - (VGG-16 (Base))	0.980392
2	Model 3: (VGG-16 (Base + FFNN))	0.980392
3	Model 4 - Simple Convolutional Neural Network...	1.000000

Model selection

The fourth model represents the most complete architecture and design. Therefore, it will be selected for evaluation on the testing dataset.

In summary, this model includes:

• CNN with a VGG-16 base model. Input size: (224,224).
• The Feed Forward Neural Network (FFNN) layers comprise 256 and 128 neurons, respectively.
• Dropout rate of 0.4.
• Adam optimizer with a learning rate of 0.001.
• Data augmentation.
• Batch size = 16.
• Shuffle enabled.

Test Performance

X_train_resized = tf.image.resize(X_train_normalized, [224, 224])
_m4_test_performance = model_performance_classification(_m4, X_test_resized,y_test)
_m4_test_performance

4/4 ━━━━━━━━━━━━━━━━━━━━ 1s 209ms/step

	Accuracy	Recall	Precision	F1 Score
0	1.0	1.0	1.0	1.0

#Prediction on testing set
y_pred_test = _m4.predict(X_test_resized)
y_pred_test = (y_pred_test > 0.5).astype(int)

4/4 ━━━━━━━━━━━━━━━━━━━━ 1s 129ms/step

print("Accuracy:", accuracy_score(y_test, y_pred_test))
print("Report:\n", classification_report(y_test, y_pred_test))

Accuracy: 1.0
Report:
               precision    recall  f1-score   support

           0       1.00      1.00      1.00        64
           1       1.00      1.00      1.00        63

    accuracy                           1.00       127
   macro avg       1.00      1.00      1.00       127
weighted avg       1.00      1.00      1.00       127

plot_confusion_matrix(_m4, X_test_resized,y_test)

4/4 ━━━━━━━━━━━━━━━━━━━━ 1s 130ms/step

Actionable Insights & Recommendations

• Continue using VGG-16 as the base model and leverage its benefits as a pre-trained model. It should always be included in future CNN designs. Resize images to (224,224) to provide VGG-16 with its recommended input size.
• It's recommended to continue seeking confirmation and mitigation of potential overfitting effects. It's possible the model is implicitly affected, even if current experiments don't show it. This investigation should be based on Model 4, as it's the most comprehensive and contains more hyperparameters that can be controlled for this objective.
• Additionally, if the dataset significantly increases in the future, it will be easier to manage with these adjustable values, favoring scalability. Consider validating for potential data leakage using other methods and by applying K-Fold Cross-Validation.

JOSUE QUIROS BATISTA